Set Max Limit on number of executors to be launched in spark plugin #101
Conversation
Codecov Report
@@ Coverage Diff @@
## master #101 +/- ##
==========================================
- Coverage 59.76% 59.76% -0.01%
==========================================
Files 95 95
Lines 5331 5348 +17
==========================================
+ Hits 3186 3196 +10
- Misses 1844 1848 +4
- Partials 301 304 +3
Continue to review full report at Codecov.
|
go/tasks/plugins/k8s/spark/spark.go
Outdated
|
|
||
| func populateSparkConfig(sparkConfig, userSparkConfig map[string]string) error { | ||
| for k, v := range userSparkConfig { | ||
| if k == "spark.executor.instances" && sparkConfig["spark.executor.instances"] != "" { |
There was a problem hiding this comment.
The current behavior of the config value is to be the default value if the users don't provide one. This will change its behavior as the max.
Let's add a a separate section for limits ?
There was a problem hiding this comment.
Updated with separate section.
go/tasks/plugins/k8s/spark/spark.go
Outdated
|
|
||
| // Spark Limits config | ||
| type Limits struct { | ||
| ExecutorLimit string `json:"executor-limit" pflag:",Max limit on number of executors allowed to be launched."` |
There was a problem hiding this comment.
nit: would suggest renaming to ExecutorCountLimit or similar just to be more explicit in-case if we want to add memory or cpu limits as well.
|
looks good. |
|
@migueltol22 is this required and are you merging it? |
TL;DR
Allows configuration to set a max limit on spark executor by specifying
spark.executor.instances:<max_limit>as part of the default spark configuration.Type
Are all requirements met?
Complete description
How did you fix the bug, make the feature etc. Link to any design docs etc
Tracking Issue
flyteorg/flyte#366
Follow-up issue
NA
OR
https://github.com/lyft/flyte/issues/